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A  Distributed  Control  System  for  the  CMU  Rover 

Abstract 

This  paper  describes  a  distributed  control  structure  developed  for  the  CMU  Rover,  an  advanced  mobile  robot 
equipped  with  a  variety  of  sensors.  Expert  modules  are  used  to  control  the  operation  of  the  sensors  and  actuators, 
interpret  incoming  data,  build  an  internal  model  of  the  robot’s  environment,  devise  strategies  to  accomplish 
proposed  tasks  and  execute  these  strategies.  Each  expert  module  is  composed  of  a  master  process  and  a  slave 
process,  where  the  master  process  controls  the  scheduling  and  working  of  the  slave  process  Communication 
among  expert  modules  occurs  as),nchronously  with  the  aid  of  a  blackboard.  Information  specific  to  the 
execution  of  a  given  task  is  provided  through  a  control  plan.  The  system  is  distributed  over  a  network  of 
processors  Operating  systetri  kernels  local  to  each  processor  and  an  interprocess  message  communication 
mechanism  ensure  transparency  of  the  underlying  network  structure.  The  various  parts  of  the  system  are 
presented  in  this  paper  and Juture  work  to  be  performed  is  mentioned. 


1  Introduction 

'fhis  paper  is  a  progress  reporton  the  CMU  Rover  Project  [1,2].  The  CMU  Rover  is  an  advanced  mobile 
robot,  equipped  wiOTSeveraHifferent  sensors,  being  developed  at  the  CMU  Robotics  Institute. 

Research  in  the  area  of  mobile  robots  can  be  divided  into  two  categories.  The  first  works  on  the  problems 
of  balance  and  locomotion  [4],  The  second  category,  in  which  this  project  belongs,  concentrates  on  the  use  of 
sensors  to  obtain  data  about  the  robot’s  surroundings,  on  the  integration  of  data  coming  from  different 
sensors,  and  on  the  development  of  strategics  for  planning  the  execution  of  tasks,  with  the  goal  of  developing 
autonomous  or  semi-autonomous  vehicles  [3,5].  The  Rover  Project  continues  research  begun  with  the 
Stanford  Cart  [1,2,3],  a  minimal,  computer  controlled,  mobile  camera  platform. 

From  the  point  of  view  of  application,  this  kind  of  research  paves  the  way  for  the  development  of 
intelligent  autonomous  vehicles  which  could  be  used  for  space  or  sea  exploration,  or  for  work  in  hazardous 
environments,  such  as  undersea  mining  and  reactor  maintenance  [6,7].  Additionally,  in  the  long  range, 
Moravcc  [2]  argues  that  research  in  mobile  robot  systems  is  an  important  challenge  in  AI  research.  The  need 
to  cope  with  dynamically  changing,  unpredictable,  real-world  environments  in  which  processing  has  to  be 
done  in  real-time,  can  lead  to  the  development  of  more  robust  and  general  AI  tools. 

In  this  paper,  we  will  give  a  brief  overview  of  the  robot’s  hardware,  describe  the  distributed  control  system 
designed  for  the  Rover,  and  present  a  set  of  expert  modules  developed  for  obstacle-avoidance  tasks. 


2  Hardware  Structure 

The  CMU  Rover  is  intended  to  support  a  variety  of  AI  research  in  the  areas  of  perception  (sensory  data 
processing  and  understanding),  control,  real-world  modelling,  problem-solving,  planning  and  related  issues. 
For  this  reason,  the  Rover  has  been  designed  with  enough  onboard  processing  capabilities  to  enable  it  to 
function  autonomously.  However,  it  is  also  connected  to  a  mainframe  system,  used  for  heavy  processing,  to 
permit  higher  overall  . performance  and  fester  response.  The  robot  also  has  several  different  sensor  systems, 
which  substitute  and  complement  each  other.  1  '  jffwKnl ' 
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The  Rover  is  cylindrical,  approximately  1  meter  tall  and  SO  cm  in  diameter,  and  is  equipped  with  three 
individually  steerable  wheel  assemblies.  Each  has  two  wheels,  and  is  powered  by  two  brushless  motors.  Each 
motor  is  controlled  by  a  dedicated  MC6805  Motor  Processor.  The  Conductor  (a  MC68000)  orchestrates  the 
individual  Motor  Processors  to  follow  a  certain  path.  The  Simulator  (another  MC6S000)  uses  feedback 
information  from  the  motors  to  maintain  a  dead-reckoning  best  estimate  of  the  robot's  position. 

The  sensors  available  include  a  TV  camera,  an  array  of  proximity  sensors  and  a  set  of  sonar  ranging  devices. 
Each  is  controlled  by  a  dedicated  MC6805  (respectively  the  Camera,  Sonar  and  Proximity  Processors).  The 
Utility  Processor  (another  MC680S)  monitors  internal  conditions  of  the  Rover,  such  as  motor  temperature  or 
battery  voltage.  Additional  processors  (all  MC68000s)  are  the  Controller,  responsible  for  the  sensor 
subsystem;  the  Blackboard  Processor,  where  information  relevant  to  several  processes  is  shared;  and  the 
Communications  Processor,  which  controls  a  remote  link  to  a  mainframe  system  (a  VAX  11/780  with  a 
high-speed  digitizer  and  an  array  processor)  (Fig.  1). 


3  Software  Structure 

3.1  Function 

The  Distributed  Control  System  provides  an  environment  in  which  the  following  functions  are 
accomplished: 

•  Parallel  and  coordinated  control  of  the  different  sensors  and  actuators,  and  handling  of  detected 
events  and  emergencies. 

.  •  Construction  of  an  internal  model  of  the  real-world  environment  in  which  the  Rover  is  operating. 

This  model  is  constructed  using  the  data  provided  by  die  sensors.  This  information  varies 
qualitatively  from  one  sensor  type  to  another,  and  may  be  incomplete  or  even  conflicting. 

•  Planning  or  acquisition  (from  the  user)  of  strategies  for  achieving  the  goals  set  to  the  Rover  by  the 
user,  and  monitoring  of  their  execution. 

A  high  level  diagram  of  these  activities  is  shown  in  Fig.  2. 


3.2  Design  Considerations 

The  design  of  the  Distributed  Control  System  was  guided  by  the  desire  to  make  effective  use  of  the  large 
potential  for  parallel  processing  existing  in  the  robot  It  should  also  integrate  gracefully  the  activities 
mentioned  in  section  3.1.  Finally,  the  control  system  should  be  flexible  and  easy  to  change  and  expand,  since 
modules  will  have  to  be  added,  tested  and  changed  frequently  as  the  system  evolves  and  the  Rover  is  used  to 
handle  different  kinds  of  tasks. 


Figure  1:  Architecture  of  the  CMU  Rover 
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Figure  2:  Main  Activities  of  the  Distributed  Control  System 
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3.3  A  Distributed  Control  System 

3.3.1  Overview 

With  the  previously  mentioned  functions  and  design  considerations  in  mind,  a  distributed  control  system 
was  designed  and  implemented.  The  system  consists  of  a  control  plan  and  an  expandable  community  of 
expert  modules.  The  control  plan  breaks  the  overall  task  set  to  the  Rover  into  a  set  of  subtasks  and  a  set  of 
constraints  (the  order  in  which  these  subtasks  must  be  executed).  Each  expert  module  is  dedicated  to  a 
particular  type  of  subtask.  One  expert  module,  the  supervisor,  dynamically  extracts  scheduling  information  on 
the  subtasks  from  the  control  plan.  The  expert  modules  communicate  asynchronously  among  themselves 
over  a  blackboard  [8].  This  blackboard  contains  information  on  the  robot's  status,  including  the  latest  update 
of  its  model  of  the  surrounding  environment,  information  abstracted  from  the  current  version  of  the  control 
plan  and  the  degree  of  progress  made  towards  completing  this  plan.  A  high  level  diagram  of  the  Distributed 
Control  System  is  shown  is  Fig.  3. 

3.3.2  Details 

Each  expert  module  is  composed  of- a  pair  of  closely  coupled  processes:  a  "master"  process  and  a  "slave" 
process.  The  master  process  keeps  track  of  relevant  information  on  the  blackboard,  changing  when  necessary 
the  status  (run,  stop,  abort,  continue)  of  its  conjugated  slave  process.  The  master  also  retrieves  necessary  data 
from  the  blackboard,  hands  it  as  input  to  the  slave  module,  and  posts  pertinent  information  generated  by  the 
slave  process  on  the  blackboard.  The  slave  process  is  the  one  actually  responsible  for  expert  work,  such  as 
monitoring  sensors,  handling  events,  controlling  actuators,  interpreting  feedback  information,  doing  path 
planning,  etc.  In  the  typical  case,  the  master  process  resides  in  the  blackboard  processor  (for  ease  of  access), 
and  the  slave  process  actually  resides  in  a  different  processor  of  the  network.  Communication  between  them  is 
maintained  through  the  use  of  a  symbolic  message  passing  mechanism  [9].  Each  of  the  expert  modules,  since 
it  works  asynchronously,  maintains  normally  an  input  processing  stream  and  an  output  processing  stream,  as 
well  as  a  special  queue  for  urgent  data  (typically  commands  related  to  abortion/suspension  of  the  process). 

The  information  posted  on  the  blackboard  has  a  structure  in  some  aspects  similar  to  the  structure  of 
messages,  being  always  composed  of  <name,  value>  pairs.  For  efficiency  and  logical  structuring  reasons,  the 
blackboard  is  subdivided  according  to  the  general  area  of  activity  to  which  the  information  is  related,  e.g., 
"movement",  "proximity  sensors",  etc.  Actual  access  to  the  blackboard  is  done  only  by  the  blackboard 
monitor,  to  insure  integrity  of  the  posted  data.  A  blackboard  scheduler  establishes  the  scheduling  of  the  master 
processes,  according  to  their  own  priorities  and  the  priorities  of  the  data  and  events  being  recorded  on  the 
blackboard. 

Necessary  to  the  overall  control  structure  of  the  Rover  is  a  control  plan.  A  simplified  example  is  shown  in 
Fig.4.  The  notation  is  based  on  [13].  Processes  can  be  specified  to  execute  in  parallel  (those  encompassed 
within  <  >  brackets)  or  sequentially  (those  encompassed  within  [  ]  brackets).  Response  to  events  is  defined  by 
the  use  of  "ON  <cycnt>  DO  <action>"  rules.  From  the  plan,  the  necessary  information  for  parallel  and 
sequential  execution  of  processes,  as  well  as  the  reaction  to  events,  is  abstracted,  and  posted  dynamically  on 
the  blackboard  as  the  plan  is  executed. 

Transparency  of  the  physical  structure  of  the  network  itself  is  obtained  through  the  underlying  support 
software  residing  in  each  processor.  Local  to  each  processor  we  have  a  real-time  operating  system  kernel , 
which  handles  interprocess  and  intcrproccssor  communication,  takes  care  of  the  internal  scheduling  of  the 
resident  processes,  and  controls  the  low-level  I/O  handling.  I/O  handlers  interface  with  the  actuators  and 
sensors  in  the  system;  other  routines  arc  responsible  for  the  translation  of  individual  low-level  actuator  and 


Rover: [Set-Goal ; 

<Go: [See-Obstacles; 

<Calcul ate-Path; 

ON  (PresPosONew-Place)  DO  <Move(New-Place) ; 

ON  (Touch)  DO 
[Stop(Move) ; 

Abort (Go) ; Run (Go)]; 
OH  (Wheel-Slipping)  DO 
[Stop(Move) ; 
Vel:=Vel/2; 
Cont(Move)] 


] 

> 

Stop-Rover]. 


Figure  4:  Example  of  a  Simple  Plan:  Moving  to  a  Goal  and  Avoiding  Obstacles 

sensor  commands  (logical->physical  translation)  and  feedback  and  sensory  information  (physical->logical 
translation),  lire  mailing  routine  constructs  messages  to  be  sent  to  other  processors,  and  receives  incoming 
messages.  Messages  have  headers  with  routing  information  (source  and  destination)  and  information ,  which  is 
in  the  form  of  <name,  value>  pairs.  The  dedicated,  specialized  modules  residing  in  each  processor  can  either 
perform  only  local  functions  (in  which  case  they  don’t  interact  with  the  blackboard),  or  can  belong  to  the  set 
of  master-and-slave  expert  modules.  The  scheduling  of  local  (in-processor)  modules  for 
execution/suspcnsion/abortion/continuation  is  done  on  the  basis  of:  a)the  inherent  priority  of  the  module; 
b)the  priority  of  internal  events,  generated  either  by  software  or  by  hardware;  c)the  priority  of  related 
incoming  messages.  The  resident  kernel  also  takes  care  of  flow  control,  buffering,  message  fragmentation, 
broadcasting.etc. 


3.4  Discussion 

The  system  presented  reflects  the  structure  of  a  community  of  cooperating  experts.  These  communicate 
asynchronously  over  the  processor  network,  generating  and  absorbing  streams  of  data.  Concomitantly,  they 
embody  a  hierarchical  model  of  distributed  computation,  where  the  arrangement  of  the  processors  can  be  seen 
as  a  tree.  This  reflects  the  fact  that  the  decision-making  model  is  partially  hierarchical:  control  decisions  arc, 
whenever  possible,  made  locally  in  the  processor  which  is  confronted  with  a  problem.  Otherwise,  the 
problem  or  data  is  broadcast  recursively  to  the  next  higher  level  of  decision  (processors  on  the  path  up  the 
tree).  Another  result  from  this  model  is  that  commands  and  data  can  exist  within  the  system  at  several  levels 
of  abstraction.  At  each  level  only  the  necessary  degree  of  detail  is  present.  Higher  levels  arc  able  to  deal  with 
the  same  information  in  a  more  abstract  form  and  do  not  become  cluttered  with  unnecessary  details. 

The  system  described  is  loosely  coupled,  since  the  rate  of  communication  between  machines,  specially  from 
the  motor  and  sensor  subsystems  level  on  upwards,  is  relatively  small.  This  results  from  the  use  of 
asynchronous  processes,  which  arc  made  possible  by  the  use  of  the  blackboard.  It  leads  to  higher 
performance  and  better  adaptability  to  dynamically  changing  conditions  [10]. 


The  blackboard  mechanism  has  been  used  by  several  researchers,  e.g.  in  speech  understanding  [8],  image 
understanding  [11].  and  tracking  of  objects  (12].  In  our  ease,  due  to  the  multiprocessing  network  and  the  need 
to  dynamically  respond  to  the  changing  conditions  of  the  real-world  environment,  the  role  of  the  "condition 
part"  of  the  Hearsay-11  Knowledge  Sources  (8]  was  expanded  to  a  more  encompassing  master/slave  relation. 
A  separate  control  plan,  integrated  into  the  overall  structure,  and  which  lacked  in  Hearsay-11,  was  also 
considered  essential,  in  our  ease.  Other  researchers  have  also  felt  the  need  to  use  separate  focusing  and 
goal-proposing  mechanisms  [11,12]. 


4  Implementation 

This  section  describes  the  present  implementation  status  of  the  project.  The  several  physical  subsystems  of 
the  Rover  have  been  tested  and  are  operational.  Final  assembly  and  testing  are  now  being  undertaken. 
Concomitantly,  one  specific  control  configuration,  composed  of  a  set  of  expert  modules,  as  well  as  the 
underlying  support  software,  was  developed.  This  set  of  modules  includes:  the  Controller,  Communication, 
Sonar,  Proximity-Sensor,  Camera,  Simulation,  Blackboard  and  Utility  modules,  which  implement  the 
corresponding  functions  mentioned  in  Section  2;  the  Movement  module  accomplishes  a  trajectory  along  a 
path  provided  by  the  Path-Planner,  the  Vision- Processing  module  extracts  visual  information  about  obstacles; 
the  Real- World- Modeller  uses  information  from  the  Obstacle-Detection  module  to  construct  an  Internal 
Model  of  the  environment;  the  Supervisor  dynamically  extracts  the  necessary  information  from  the  Control 
Plan,  while  the  Defaulter  fills  in  the  details;  the  Watcher  prevents  dangerous  situations  from  happening,  and 
the  User-Interaction  module  permits  communication  with  a  human  user.  These  modules  have  been 
implemented  and  tested  in  a  simulated  environment.  The  control  system  proved  very'  flexible  and  adequate, 
and  is  now  awaiting  in  silo  testing,  running  the  Rover. 


5  Conclusion 

Although  the  development  of  such  a  complex  piece  of  hardware  and  software  is  very  demanding,  it  has 
already  brought  many  new  insights  in  various  different  areas,  such  as  control,  planning,  interaction  with  the 
environment,  benefits  that  arise  from  the  use  of  a  processor  network,  etc.  The  Distributed  Control  Structure 
presented  fulfilled  the  initial  design  goals,  and  greater  experience  with  it  will  be  gained  when  exercising  the 
Rover  in  different  kinds  of  tasks. 

Future  work  to  be  done  includes  the  graceful  integration  of  information  coming  from  the  sensors  into  a 
more  sophisticated  Internal  Model,  the  development  of  a  more  elaborate  planning  system,  and  research  in 
automatic  knowledge  acquisition  and  learning. 
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